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ABSTRACT 



In a problem-based curriculum, emphasis is placed on the 
groups in which students learn to analyze problems and to contribute to the 
solution of a problem. This paper describes an instrument that aims to 
measure individual group performing and presents some psychometric results. 
Reliability and validity were studied with 240 students in groups of 9 with 1 
tutor. The instrument appears to be reliable, with coefficient alpha varying 
from 0.84 to 0.92, and it seems that it is a valid measure for 
characteristics that are important in a problem-based curriculum: (1) 

participating in discussions; (2) explaining things to other students; and 
(3) a broad study approach. (Contains 2 tables and 11 references.) (SLD) 
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ABSTRACT 

In a problem-based curriculum emphasis is given to groups. In a group students learn to 
analyze problems and to contribute to the solution of the problem. Students also learn to 
cooperate. The paper describes an instrument which aims to measure individual group 
performing. Also some psychometric results are presented. The instrument turns out to be 
reliable (coefficient alpha varies from 0.84 to 0.92) and it seems that the instrument is a valid 
measure for characteristics which are important in a problem -based curriculum: 
participating in discussion, explaining to other students, and a broad study -approach. 



INTRODUCTION 

Education is generally seen as a process of transmission of (Barrows, 1983; Bruhn, 1992; 

Schmidt, 1993; Van der Vleuten & Wijnen, 1990). There are teachers who have knowledge at 
their disposal. They passes this onto a student. The student receives knowledge from the 
master. One could also approach education from the other direction, beginning with the 
student who wishes to acquire knowledge and who comes to an education institution, which 
provides the opportunities and tools for this purpose. These could be books or articles, audio - 
visual aids, or practical attachments, fellow -students or teachers. The teacher is, in this case, 
no longer the person, who provides students with ready -made materials in the form of 
lectures and notes. Instead the teachers become the student's guides, who stimulate the 
learning process and, where necessary, indicate the direction that it can take. As a result, 
education is no longer, as is generally the case, teacher- centered, but student-centered 
(Van Berkel, 1990; Norman & Schmidt, 1992). 

The process of problem- based learning starts with an analysis of the problem that is offered 
by (one of) the assignment(s) in the block boo k. Students discuss in a group session the nature 
of the problem and think of as much knowledge that is possibly related to the subject. 
Students utter freely the considerations and ideas/inventions that come into mind as free 
associations during the brainstorming phase. After that a critical evaluation takes place of 
the proposed knowledge items and points of consideration, the group tries to weigh the 
propositions on their relevance and importance both in relation to the specific problem 
considered and to the knowledge domain where the problem was taken from, and position 
the not discarded topics into a structured frame of reference. At the end of this first group 



'Paper presented at the AERA-conference (San Diego) on April 17th 1998 




2 



session the group determines which aspects they want to investigate more deeply. These 
aspects are recorded in 'learning objectives', stated in the form of questions that guide the 
self study activities that take place between the first and second group session. All group 
members occupy themselves individually or in cooperation with other students with the 
connected study topics with the help of handbooks, literature, etc. At the following group 
meeting an inventory is made of the results of this study. This will often lead to group 
discussion. Two members of the group can, for example, arrive at d ifferent answers to the 
same question, or it can be that the material that has been studied, does not provide 
sufficient clarity. Being critically active with the information one has found, increases the 
depth of insight into the problem (Schor et al„ 1997; Norman & Schmidt, 1992; Schmidt & 
Moust, 1995). If the group determines that they have made sufficient progress with a problem, 
they start with the next assignment in the block book. 

Besides content -oriented learning goals, social skills and attitudes are also essential in 
preparing for a profession, e.g.: active listening to students, acquiring professional attitude in 
problem settings, verbal participating in a discussion, playing the role of chairman, and 
reporting in the group about individual study activities (Van Berkel, 1990). This kind of learning 
objective is not evaluated in the end- of-course test. The faculty of Health Sciences at the 
Maastricht University developed an instrument for qualifying students' performing in tutorial 
groups. The objective of this study is to evaluate the instrument. 



INSTRUMENT 

At the end of a course the tutor judges the adequacy of students group 
performing/participation with regard to the following criteria for effective performing in 
problem-based group learning. 

1 Contribution to the analysis of the problem and to the statement of the learning- goals; 

2 Keeping one's agreement; 

3 Contribution to the discussion concerning the collected data; 

4 Fulfilling the role of chairman in a group session; 

5 Contribution to the promotion of the group process. 

The judgement of the tutor is a dichotomous one: a student receives or receives not a so - 
called 'PTG-point' (Performing in Tutorial Groups -points). There is a limit to the number of PTG- 
points a tutor may distribute: the maximum number of points may not exceed the number of 
group members divided by two. The faculty decided for the limit because a pilot experience 
of the method demonstrated that tutors are too generous: they donated almost every 
student in the group with a point, because tutors found it hard to disqualify students. This is not 
an unusual finding. In face-to-face judgements there is a consistent lack of discrimination. So 
it was compulsory for tutors to use normative based judgements, although it is presume d that 
in this situation criterion-based judgements are favorable: if each student performs well 
according to the criteria, each student deserves a PTG -point. On the other hand, it turns out 
that the mean percentage of distributed PTG -points was 31%, in st ead of 50%. So in practice 
the tutors judged criterion-based. 

Because there are six courses a year, the maximum number of points a student can receive in 
a year, is six. To a certain degree, the collected points can compensate the score on the 
end-of-unit test (see Van Berkel, 1995). 



PROCEDURE 



To determine the psychometric qualities of the instrument an evaluation study was initiated. 

In a parallel judgment procedure all the group participants, tutor and students, were asked to 
choose the three fellow students who to their opinion were the best candidates to receive a 
PTG-point, and state their names in order of preference. Next these three students had to be 
rated on a list of characteristics that constitutes an operationalization of the five judgment 
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taken from several studies regarding problem-based learning (Schmidt, 1993; Norman & 
Schmidt, 1992; Cariaga-lo et al., 1996). 

(Insert scheme 1 here) 



ANALYSIS 

The psychometric analysis applied to these data consisted of two parts. First, the inter-rater 
reliability of the parallel PTG-judgment was calculated by computing coefficient alpha. 
Second, the validity of the judgment procedure was determined by its predictability from 
behavior characteristics. 

Reliability 

The population comprised 240 students. Because this population is randomly assigned to 
tutorial groups, all the judgment panels (i.e., each nine fellow students and a tutor), can 
legitimately be regarded as replications, just as is the case with parallel tests. So, it is justified 
to calculate alpha reliability on a 'test' matrix, containing ten items for each judged student in 
the population (Thorndike, 1982). Alpha coefficient was 0.88, a remarkable high value. This 
result means that there is a strong agreement among students on which students can be 
rewarded with a PTG-point. We also computed the correlation between the rating of the 
tutor and the ratings in the parallel judgment procedure used in t he evaluation study. The 
correlation was 0.87, indicating that the judgment procedure in this study is highly 
representative to the qualifications that students assign in the formal context. 

Validity 

Predictability of group performing judgment was determi ned against two criterion variables. 

1 . receiving a PTG-point by the tutor, (scored: yes/no) 

2. the number of received PTG -points by the students. 

The prediction of these two criteria was investigated by means of discriminant -analysis 
(criterion 1) and regression analysis (criterion 2). 

The discriminant function, determined by entering all twenty behavior characteristics of the 
questionnaire, delivered a correct classification for 81% of the students on the criterion: yes or 
no a PTG-point. (When the PTG-points are randomly assigned the percentage correct 
classification is of course 50). 



(Insert table 1 here) 

.The items which show the highest correlation (=> 0.30) with the discriminant function are the 
items 01 (r = 0.62), 20 (-0.51), 16 (0.47), 03 (0.41), and 14(0.31). 

The regression analysis (Method: Enter) indicated that the twenty items explain 56% of the 
variance in criterion 2 (see table 2). 



(Insert table 2 here) 
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The items that showed to be the most important in the discriminant analysis related to 
criterion 1 , were also the ones that contributed significantly to the regression results for 
criterion 2: item 01 (Beta = 0.39), 16(0.27), 14 (0.22), 03 (0.17%), except item 20, that received 
a beta value of only 0.05. 



CONCLUSION AND DISCUSSION 

Only five of the twenty behavior characteristics contribute significantly to a prediction of PTG- 
ratings (regression analysis), although most of the items ideally are to be regarded as highly 
relevant indicators of the five judgment criteria. Clearly the most important factor that 
determines a studentsD chance on receiving a PTG -qualification, is his/her verbal dominancy 
in the group (items 01, 20), but some important qualitative aspects of group performing 
likewise contribute substantially (items 03, 14, 16). The activities referred to in these items, 
mainly concern the presentation of acquired knowledge (items 03, 16). 

Interrater-reliability appears to be high. Apparently it is evident to most group members which 
of them should be assigned the PTG-point. This is not a surprising result, looking at the best 
predictor of the PTG-score, the amount of students' talking in the group, because talking as 
such is a clearly observable characteristic. But it appears that also other, less clear- cut 
aspects play a role of significance in the valuation of a student by other participants as 
'contributing to group effectiveness'. With respect to these characteristics (like 'uses a broad 
perspective') reasonably much more divergence in position could be expected between 
judges, resulting in less homogeneity of their qualifications. 
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Scheme 1 : Characteristics of group performing behavior 



/fern Characferisf/c 

01 To Iks little/ much 

02 Produces useful ideas 

03 Explains to other students in the group 

04 Asks for explanations 

05 Poses questions to the tutor 

06 Reacts to questions/suggestions of the tutor 

07 Makes corrections on other group members' contributions 

08 Does not just accept information as being true 

09 Watches over the discussion in general 

1 0 Is prepared to fulfill the role of chair 

1 1 Takes position with regard to the process of group cooperation 

1 2 States opinions about the advisable task approach 

1 3 Works hard during self study time 

1 4 Uses a broad perspective in his/her study approach 

1 5 Always completes his/her home work 

1 6 Supplies an useful contribution to reporting activities 

1 7 Gives the impression to be interested 

1 8 His/her choice of activities is ruled by weighing considerations of usefulness 

1 9 Poses high demands on learning results 

20 Seems to be inhibited to say a lot 



Table 1 


Correlations between characteristics of group performing behavior and the 
discrimination function 


Item 


Characteristic 


Correlation 


01 


Talks little/much 


0.62 


02 


Produces useful ideas 


0.23 


03 


Explains to other students in the group 


0.41 


04 


Asks for explanations 


0.10 


05 


Poses questions to the tutor 


0.08 


06 


Reacts to questions/suggestions of the tutor 


0.23 


07 


Makes corrections on other group members’ contributions 


0.01 


08 


Does not just accept information as being true 


0.10 


09 


Watches over the discussion in general 


0.14 


10 


Is prepared to fulfill the role of chair 


0.15 


11 


Takes position with regard to the process of group cooperation 


0.28 


12 


States opinions about the advisable task approach 


0.17 


13 


Works hard during self study time 


0.29 


14 


Uses a broad perspective in his/her study approach 


0.31 


15 


Always completes his/her home work 


0.21 


16 


Supplies an useful contribution to reporting activities 


0.47 


17 


Gives the impression to be interested 


0.18 


18 


His/her choice of activities is ruled by weighing considerations of usefulness0.08 


19 


Poses high demands on learning results 


0.16 


20 


Seems to be inhibited to say a lot 


-0.51 
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Table 2: Multiple regression analysis (method Enter): Getting a PTG-point (independent 

variable) and characteristics of group performing behavior (dependent 
variables) 



Item 


Characteristic 




Beta 


Significance 


01 


Talks little/much 




0.39 


0.00 


02 


Produces useful ideas 




-0.17 


0.04 


03 


Explains to other students in the group 




0.17 


0.04 


04 


Asks for explanations 




-0.09 




05 


Poses questions to the tutor 




0.06 




06 


Reacts to questions/suggestions of the tutor 


0.01 




07 


Makes corrections on other group members ' 










contributions 




-0.15 


0.09 


08 


Does not just accept information as being 


true 


-0.07 




09 


Watches over the discussion in general 




0.13 


0.07 


10 


Is prepared to fulfill the role of chair 




-0.01 




11 


Takes position with regard to the process 


of 


-0.11 






group cooperation 








12 


States opinions about the advisable task 










approach 




0.01 




13 


Works hard during self study time 




-0.04 




14 


Uses a broad perspective in his/her study 










approach 




0.22 


0.00 


15 


Always completes his/her home work 




0.04 




16 


Supplies an useful contribution to reporting 








activities 




0.27 


0.01 


17 


Gives the impression to be interested 




0.00 




18 


His/her choice of activities is ruled by 










weighing 




-0.04 






considerations of usefulness 








19 


Poses high demands on learning results 




0.14 


0.09 


20 


Seems to be inhibited to say a lot 




-0.05 





Multiple R: 0.74 
R-square 0.56 
Sign. F: 0.00 
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